feat(media): a converter receives a source handle, never the file's bytes (D104) - #344

Problem

A text file fits in memory and a video does not. Under D65's convert(bytes, mime, hints) contract, the client, the HTTP process, the object store, and the converter each materialize a complete copy of the same file. Two concurrent large ingests can exhaust a worker that would have processed either one alone. That is an availability property, not an optimisation — and it is the reason no media route can be built on the current contract.

Scope

• SourceHandlePort — bounded reads of one immutable, already-hashed source;
• ObjectSourceHandle — the adapter over any ObjectStorePort;
• open_stream / read_range on ObjectStorePort, implemented for the local filesystem and MinIO;
• D104 in decisions.md, design in plan/designs/media_design.md §2.1.

Nothing consumes the handle yet. No converter signature changes here and no route behaviour changes. This is the seam a media route needs to exist at all.

Important decisions

Four reads, every one of them bounded. Stream in bounded chunks; read a half-open range [start, end); materialize to a temporary local file for decoders like FFmpeg that need a seekable path; or read the whole thing into memory after naming the limit. There is no operation that reads a source of unknown size — the hazard was never bytes in memory, it was bytes in memory without anyone having decided how many.

A read ceiling, because naming a large limit is still naming one. read_range(0, byte_size) is a whole-file read with extra steps, so a handle can carry max_read_bytes and refuse any single read above it. Without a ceiling the property is ergonomic friction; with one it is enforced. The design says which it is rather than overclaiming.

A stream is a resource, not just bytes. Underneath is a file descriptor or an HTTP body, so open_stream is a context manager released at block exit. A bare iterator would leave an early-stopping caller holding it until garbage collection, and GC timing is not a lifetime contract.

Reads never come back short, and length is verified alongside hash. A range returns exactly what was asked for or raises — a parser handed a truncated header cannot tell it from a valid one. Bytes can also hash correctly while being described by a wrong size, which would let materialization and range reads disagree about the length of one source.

Half-open ranges match every other interval in the system (D65 locators), so nobody has to remember which end is inclusive. S3 ranges are inclusive on both ends, so the exclusive end becomes end - 1 on the wire — the conversion lives in the adapter so the port stays consistent regardless of provider.

Bounds live in materialization, since that is the operation that can actually exhaust a host:

• the caller declares what it can afford, and an oversized source is refused before any byte moves — one integer comparison, not a filled disk;
• written bytes are verified against the recorded content hash. An immutable object cannot legitimately change, so a mismatch is corruption or a wrong key, never retryable. Without this check a truncated read becomes a short document that looks complete — the silent-loss failure D65's coverage rules exist to prevent;
• the recorded size is a claim rather than a guarantee, so the streaming write also counts what it writes and refuses a source exceeding both the declaration and the accepted bound; and
• the temporary file's lifetime belongs to the handle, so a route that fails mid-decode cannot leak the file it asked for.

One contract, both deployment shapes. A self-host directory tree and a cloud object store present the same three operations, so no route branches on where bytes live.

Sources inspected

Base origin/main 7c41fdb8. D38 (router), D57 (Markdown coordinate system), D61 (port boundary), D65 + plan/designs/media_design.md §2 (media routes and the converter contract). read_bytes is retained deliberately — every shipped route converts small text and has no reason to stream.

Review

Double-reviewed by two independent reviewers. Both returned BLOCK and converged on the same two blocking defects; the second found five more, including one aimed squarely at this PR's own claim.

Round 1 — teardown used unlink + rmdir, which raises directory not empty the moment a decoder writes a sidecar beside the source, leaking the directory and masking the route's own exception. And reads could come back short: LocalFS truncated past EOF, MinIO leaked botocore.ClientError on a 416 (a provider type in a port D61 keeps provider-free), and the handle checked only the recorded size, never what actually arrived.

Round 2 — read_range(0, byte_size) made the "no whole-file read" claim false, so the ceiling was added and the docs corrected; open_stream leaked descriptors on abandonment; the streaming guard fired only after a full 1 MiB chunk had moved for a 100-byte allowance; length was never verified against the record; SourceIdentity.mime duplicated an authority the converter signature already owns. Two tests also asserted less than their names claimed — "refused before any bytes move" only checked that no file survived — and now use a counting store asserting bytes actually served.

Concurrent materializations still have no aggregate temp-disk budget. That is worker-level admission, and it is documented as a stated boundary rather than quietly left.

Validation

• ruff format --check and ruff check clean;
• pyright — 0 errors;
• pytest — 1303 passed, 685 skipped;
• 29 new tests across test_source_handle.py and test_minio_store.py: ordered streaming, half-open ranges, degenerate and past-the-end ranges, cleanup on success, on exception, and with a sidecar present, refusal before any byte is served, the understated-size guard bounded to ≤101 bytes moved, hash mismatch, size mismatch, the read ceiling, zero-byte sources, temp_root=None, concurrent materializations, stream release after abandonment, and the MinIO wire translation asserting [2, 5) reaches S3 as bytes=2-4.

One pre-existing failure, test_sdk_pushes_lineage_metadata_to_e0, reproduces on base commit 7c41fdb8 with these changes stashed: mimetypes on this machine does not resolve .md, so the SDK guesses application/octet-stream. Not introduced here and not fixed here.

Extending ObjectStorePort required adding the two reads to three test fakes and raising the port-inventory count to thirteen.

Limitations

The handle has no consumer yet — the D65 converter signature still takes bytes, and changing it is the next step rather than part of this PR. The trusted structural probe (container, tracks, codecs) is not here. No video route exists and none is claimed.

Follow-up

1. Move the converter contract onto the handle, route by route.
2. The trusted probe, as the other half of the D65 companion amendment.
3. Managed upload/resume, which is cloud-side and has its own contract to write.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XTv3baR8R5EgEGFePkQMjK